This is so strange, my change is not even related to e2e test for some functionality, but all the e2e test keeps failing

Metrics Comparison Summary

summary_metrics_snapshot_kafka

📊 Metrics Diff Summary

Total Changes: 3

• 🆕 Added: 0 metrics
• ❌ Removed: 0 metrics
• 🔄 Modified: 3 metrics

🔄 Modified Metrics

• otelcol_kafka_exporter_records
  • Added variants: 1
  • Removed variants: 1
• otelcol_kafka_exporter_messages
  • Added variants: 1
  • Removed variants: 1
• otelcol_kafka_exporter_bytes_uncompressed_bytes
  • Added variants: 1
  • Removed variants: 1

summary_metrics_snapshot_cassandra

📊 Metrics Diff Summary

Total Changes: 53

• 🆕 Added: 53 metrics
• ❌ Removed: 0 metrics
• 🔄 Modified: 0 metrics

🆕 Added Metrics

• http_server_request_body_size_bytes (18 variants)
• http_server_request_duration_seconds (17 variants)
• http_server_response_body_size_bytes (18 variants)

summary_metrics_snapshot_elasticsearch

📊 Metrics Diff Summary

Total Changes: 73

• 🆕 Added: 0 metrics
• ❌ Removed: 73 metrics
• 🔄 Modified: 0 metrics

❌ Removed Metrics

• jaeger_storage_latency_seconds (18 variants)
• jaeger_storage_requests_total (1 variants)
• rpc_server_duration_milliseconds (18 variants)
• rpc_server_requests_per_rpc (18 variants)
• rpc_server_responses_per_rpc (18 variants)

summary_metrics_snapshot_cassandra

📊 Metrics Diff Summary

Total Changes: 53

• 🆕 Added: 53 metrics
• ❌ Removed: 0 metrics
• 🔄 Modified: 0 metrics

🆕 Added Metrics

• http_server_request_body_size_bytes (18 variants)
• http_server_request_duration_seconds (17 variants)
• http_server_response_body_size_bytes (18 variants)

➡️ View full metrics file

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.63%. Comparing base (c1f0f36) to head (b028252).
⚠️ Report is 4 commits behind head on main.

@@             Coverage Diff             @@
##             main    #7449       +/-   ##
===========================================
+ Coverage   33.68%   96.63%   +62.95%     
===========================================
  Files         202      377      +175     
  Lines       12986    23089    +10103     
===========================================
+ Hits         4374    22312    +17938     
+ Misses       8269      592     -7677     
+ Partials      343      185      -158     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hi @yurishkuro, I don't know why some metrics are changed(may be e2e test is non deterministic at some parts). But this pr should be good now.

yes we need to fix this. In Kafka e2e tests we use randomized topic names and the topic is added as a label to the metrics, causing the differences. We may need to add some sanitization to the metric dump steps to remove random suffixes in metric labels.

Don't know about cassandra, it's possible it's the same reason, a randomized namespace inserted as labels.

Btw it would be useful to include a sample of diffs, e.g. 5-7 lines, in the report. We can expect the number of diffs to be relatively small across PRs, and it's much easier to see the report directly than to download and unpack the artifacts.

Btw it would be useful to include a sample of diffs, e.g. 5-7 lines, in the report. We can expect the number of diffs to be relatively small across PRs, and it's much easier to see the report directly than to download and unpack the artifacts.

Hi @yurishkuro,
Make sense, I can do it in a separate pull request, thanks for your feedback!